[improve] add read rate limter for compaction#3976
[improve] add read rate limter for compaction#3976HQebupt wants to merge 1 commit intoapache:masterfrom
Conversation
| long compactSize = meta.getTotalSize() - meta.getRemainingSize(); | ||
| compactionReadByteRateLimiter.acquire((int) (compactSize)); | ||
| compactEntryLog(meta); |
There was a problem hiding this comment.
If I understand correctly, when the compact is in progress, if compactionRateBytes is configured, the compact will be limited when writing the disk. Since read and write are processed strictly serially, the next read needs to wait for the write operation to complete. Does it mean that read is also limited to compactionRateByBytes？
I suspect that your problem here is because the disk read rate is too fast caused by getting entryLogMeta from entryLogFile, you can look at this PR: #2963
@hangc0276 Please help take a look. Thanks.
There was a problem hiding this comment.
@HQebupt Can you search in your bookie service log to see if there is the following log when the read IO is high: “Extracting entry log meta from entryLogId”
There was a problem hiding this comment.
@wenbingshen It did not help after I cherry pick #2963. We can discuss it when you are available tomorrow.
// Set the rate at which compaction will readd entries. The unit is bytes added per second.
compactionRateByBytes=31457280
|
It works, thank you @wenbingshen @hangc0276 @StevenLuMT I closed the PR. |
4 participants
Motivation
The purpose of this feature is to provide more flexibility and control over the compaction process in the Bookkeeper server by allowing users to configure the read rate of compaction according to their specific requirements.
It adds support for configuring the read rate of compaction in bytes per second.
In our production environment of Bookkeeper, we have set compactionRateByBytes=30M. However, we have observed that it only limits the write speed to disk and does not limit the read speed from disk. In our environment, the continuous saturation of disk IO for reads in compaction is causing issues with data consumption from the disk.
Changes
GarbageCollectorThreadclass, a new fieldcompactionReadByteRateLimiterof type RateLimiter is added to control the read rate of compaction.ServerConfigurationclass, a new configuration propertyCOMPACTION_READ_RATE_BY_BYTESis defined to specify the read rate of compaction in bytes per second.Master Issue: #8